Python 2 EOL Roadmap?

[fake-name]

LinuxCNC includes a lot of python components, and as everyone is probably aware, Python 2 is EOL in a little more then 2 years.

What's the roadmap for the python 3 migration?

It might be a decent idea to start requiring that people who touch python files make them support either platform. It's completely possible to write python code that works in both versions.

[SebKuzminsky]

I would love an effort to make our python code 2/3 compatible.

Are there tools in the python world that can identify 2/3 compatibility problems in python programs and modules? Maybe through static analysis? "Flake" maybe?

If so we should add build-time tests to run them; for now the test will just warn on failures, then once we have full 2/3 compatibility we'll switch it to fail the build on failures.

[SebKuzminsky]

pylint --py3k seems like an option. Here's the output for axis:
pylint-axis.txt

[gmoccapy]

I agree, that we should go the way to write code for both python versions. A good way to start is also the page:

Python future

I will check also on pylint

Norbert

[KurtJacobson]

The python code will be easy to convert to work with both Python 2 and 3, the part I am not so sure about is the C/C++ python bindings. I think boost python is used for those. We probably will have to add a switch to compile for one or the other version of python. Ideally the bindings would work with both, but I am not sure how feasible that is.

[fake-name]

I would love an effort to make our python code 2/3 compatible.

Note that you don't have to have the same code work for both. There are many packages written for Py2k that support py3k via 2to3, which is invoked during package installation. Basically, they're written with a little care so that the automatic translation allows them to be converted to python3 without issue.

[c-morley]

@fake-name as Kurt mentioned, the key problem is the c/c++ binding code.
It seems that we really need to make a choice at one point and change everything all at once.
And python touches almost everything in linuxcnc! Huge job.
Nice job there Python :(

[SebKuzminsky]

Here's somebody else's experience converting an old, big python2 codebase to python3: https://medium.com/@boxed/moving-a-large-and-old-codebase-to-python3-33a5a13f8c99

There might be something helpful for us to glean from this....

[fake-name]

FWIW, I'm rewriting axis.py for work things. Primarily, I'm doing cleanup (holy cow it's the worst python I've ever worked on), but it should be basically 2/3 compatible when I'm done.

It actually resulted in https://github.com/fake-name/autoflake/blob/master/autostar.py, (pr here), which is a kind of cool thing that tries very hard to fix all the from x import * mess in a python file, by doing some crazy ast hijinks to figure out which packages items not defined on the local scope come from, and rename them to their actual full names.

Actually, I'm unclear why axis.py involves tcl at all, but that's a different discussion.

[c-morley]

SebKuzminsky: An interesting read there. I see nothing of python extensions or python embedding unfortunately, which of course is the hardest part. It's a good reference though.

[jepler]

@fake-name It's great that you are taking an interest in working on axis and on the externally imposed python2 time bomb that our project faces. Thank you.

But even as you have the urge to rewrite axis.py wholesale (who doesn't? It's more a question of having the energy to do so), please keep in mind how you'll be able to ultimately create that as a pull request where the individual commits are reviewable. This is particularly important since we don't have any tests of GUI components of LinuxCNC, like AXIS. (Unless you're going to fix that too. 😉) A single commit in which the whole structure and design of axis.py is upended is probably unreviewable and is likely to languish indefinitely as a pull request.

The way lots of direct access to Tcl/Tk is mixed with Python/Tkinter does have an historical motivation: Long ago at my $DAY_JOB, I created a GUI builder for Tcl, which I got permission to release as an open source product. (it didn't gain any users and the open source version is unmaintained) A never-realized goal was for programmers to be able to reconfigure the AXIS UI by modifying it in the GUI builder. AXIS never achieved this goal, and eventually the GUI builder files were renamed to ".tcl" and when it felt expedient, developers (mostly me) wrote code in Tcl instead of in Python. Coming from my particular $DAY_JOB's background, mixing Python and Tcl like this doesn't even feel weird; but I know it's not common practice in widely published or read open source Python projects.

[fake-name]

But even as you have the urge to rewrite axis.py wholesale (who doesn't? It's more a question of having the energy to do so), please keep in mind how you'll be able to ultimately create that as a pull request where the individual commits are reviewable.

My primary focus here is to get axis into a state where I can extend it as an internal testing tool for work-related things. The rework is basically only because the code is impenetrable as-it-is, due to the complete lack of structure.

Anyways, this would never be a single commit, but rather a series of incremental changes which maintain functionality with incremental refinement.

[rene-dev]

I had a go at this today, and after some help from @jepler stuff started to build.
There is a problem with a dependency for halcompile:
http://theory.stanford.edu/~amitp/yapps/
there is a python 3 compatible version, but its not available in Debian.
I will create a pull request later with what I have done so far.

[rene-dev]

my work in progress branch is here: #416

[loganpowell]

Just adding something from the source on the topic:
https://docs.python.org/3/howto/cporting.html#cporting-howto

from:
https://docs.python.org/3/whatsnew/3.0.html

also this: http://python-future.org/automatic_conversion.html

source

[c-morley]

I'll add this proof-of-concept branch of using cython for HAL bindings.
If we also had py3 binding for the motion controller, then screens could use python 3.
Then remap would probably be the next big problem.

https://github.com/LinuxCNC/linuxcnc/tree/python3_cython

[rene-dev]

I pulled a few python3 fixes into master, and configure and and python syntax checks are now working.
Next step is to fix the python bindings. anyone want to try?
./configure --with-python=python3 --with-boost-python=boost_python-py35

[rene-dev]

It does compile now, and:
Runtest: 227 tests run, 183 successful, 44 failed + 0 expected
most of the fails are due to the hal python module not working.
the buffer protocol stuff changed, and hal is currently only a dummy module that requires some more time. No gui works. There are a lot of type and import problems.
I am not sure it will be possible to support 2 and 3, but I dont think we need to.
python2 will be EOL in January, and python3 is already 11 years old, and every modern OS comes with it. But I will look into using the preprocessor to make the modules 2 and 3 compatible.

[gibsonc]

Adding my small contribution for those keen on experimenting / assisting

I have been keen to learn C++ for a while and I can find my way around Python so I decided to give this a bash and see how far I get. This is also the first time I have used git or contributed to any project, so please excuse any ignorance.

I have forked 2.8 to https://github.com/gibsonc/linuxcnc/tree/2.8
I have made no attempt to support Python 2 and 3 - it simply spirals out of control with all the if / else constructs. It all compiles now except for halmodule.cc due to the depreciated buffer protocol.

https://docs.python.org/3/c-api/objbuffer.html

I still need to get my head around that. As a result a lot of the tests currently fail.

On Mint 19
pip3 install yapps
./configure --with-python=python3 --with-boost-python=boost_python-py36

[gibsonc]

Just an update for the curious. It is a lot of work, mostly repetitive, but more things are starting to work now. Thanks to @jepler for his assistance. I cut down on another ten failed tests this evening and with what he taught me I think I will be able to resolve a few more quite soon. I am no coder but I am learning fast and if I can learn so can anyone else :)
The only thing that worries me is how we will merge again, but that is a problem for later down the line.

[strufkin]

if you interested in i've ported C code and all features i needed worked for me (it was some time ago, hoping this helps you, unfortunately it is a bit behind current master, let me know if i could help here, the only thing i've disabled was related to shmbuffer_procs also found in old mail lists that it is not being used by anyone, also there are no tests for that interface, so seem it needs some discussion )
https://github.com/strufkin/linuxcnc/commits/py3-work

[gibsonc]

Oh man - how I wish I had seen your work before I started. Nothing is wasted however - I have learned a lot from practically nothing. I didn't know about py3c - it makes all this Bytes / Unicode pain go away that I am currently experiencing. I fix it, but it seems to be by chance and that is never a good thing. Ideally the code must check for the types and return the correct type.
It would seem a few of us are duplicating effort in different areas. How can we better pool our efforts and move forward?

[saxa]

Make a paralel branch and commit faster your work to it :)

[andypugh]

Maybe we should all choose one branch and all work on it?

Maybe this one, as it is already in the LinuxCNC repository?
https://github.com/LinuxCNC/linuxcnc/blob/dgarr/py3/PY3_README
But that is only a suggestion.

[spiderdab]

Hi, I just installed Debian Buster on a VM and compiled dgarr/py3 branch as RIP. No big issues to install. the Buster I downloaded was the '10.2 amd64 net install' iso, and python 2.7 was the default version, so I had to 'update-alternatives' to set python 3.7 as the default version.
I was looking to the python3 porting of lcnc, and came accross this repo which seems to be the more advanced in the migration. I personally needed only hal to be python3 compatible, but if I can I would like to help here with the code (if I'm able to..) or at least doing tests.
Now it is installed in a VM because I'm developing a GUI in GTK3 which uses hal to setup some stepper generators on a mesa 7i96 (..to be changed after for more generators..) and will transfer that on a working machine after it works.

I can report that importing hal works with python3 and also axis seems to work (sim mode only..)

[rene-dev]

I am currently working on this, picking the best of the attempts that have been made, and getting them up to speed with master.

[rene-dev]

status update:
Im picking the best parts of the branches that have already been started by other people, and updating them up to current master.

working:

• Axis
• All but a few remap related tests pass

not working:

• Any gtk2 UI, gmoccapy, gscreen, touchy, ...

not tested:

• qt5
• actually running a machine

This is the roadmap I came up with:

• Merge all python script related stuff into master and make them 2/3 compatible
• add a compile check, to make sure they stay compatible: Add pycheck (Python check) Makefile target #807
• use py3c to make the bindings 2 and 3 compatible: https://py3c.readthedocs.io/en/latest/guide.html

py3c is not available, on wheezy, so either drop support for wheezy(#743), or see if it still works for python2 if you bundle py3c.

currently the roadblock is gtk2. not all gtk2 bindings are available in python3.
This is also an issue on buster, see #809
I started porting gscreen to gtk3, but the problem is that you cant reliably create a opengl context in gtk3:
https://gitlab.gnome.org/GNOME/gtk/issues/64
https://gitlab.freedesktop.org/mesa/mesa/issues/113

[rene-dev]

gtk3 gtkglarea problem has been fixed, waiting for them to accept the PR
https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/1790

[rene-dev]

done, master now build with python3, and all but one tests pass.
I will open individual issues for the new problems.
Thanks to @dngarrett and @strufkin for their work on the python modules, my work is partly based on their work.
Thanks to @jepler for helping with the build system and various python problems.

[zultron]

Huge congrats to @rene-dev and the others who accomplished this, and huge thanks, too: we're borrowing from your work over at machinekit/machinekit-hal#114 . You all rock.

[jallwine]

When I download the Raspberry Pi 4 image, the linuxcnc python module seems to only be available when running python 2. Does that image not have the latest, or has that not been addressed yet?

[andypugh]

LinuxCNC 2.8 still uses Python 2. Master / 2.9 has made the switch to Python 3, but not everything is working yet.

[jallwine]

Gotcha, thanks @andypugh!